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CLAIMS 

1 . A method of processing input data defining (i) the positions of features 
in a sequence of images of at least one object which represent features on the 
object, (ii) an estimate of a respective camera projection for each image 
defining the projection of points on the object into the image, and (iii) 3D 
feature points comprising estimates of the positions in three-dimensions of 
features on the object represented by the features in the images, to generate 
output data defining further estimates of the camera projections, the method 
comprising processing respective subsets of the images in the sequence by: 

calculating an error for the camera projections of the images, in the 
subset by projecting selected 3D feature points into the images in the subset 
using the camera projections of the images in the subset and determining the 
distance between the positions of the projected points and the positions of the 
corresponding features in the images; and 

calculating changed positions for at least some of the selected 3D 
feature points and changed camera projections for the images in the subset by 
using the positions in the images in the subset of the features which 
correspond to the selected 3D feature points to determine changed positions 
for at least some of the selected 3D feature points and changed camera 
projections for the images in the subset which reduce the calculated error; 

wherein the respective subsets of images are selected sequentially from 
the sequence so that each image in the sequence is processed in a subset at 
least once. 
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2. A method according to claim 1, wherein the changed positions for the 
3D feature points and the changed camera positions are calculated by reducing 
the calculated error by performing processing in accordance with a 
minimisation procedure to change iteratively the positions of at least some of 
the selected 3D feature points and the camera projections of the images in the 
subset and to calculate an updated error by projecting the 3D feature points 
from their changed positions into the images in the subset using the changed 
camera projections of the images in the subset, and determining the distance 
between the positions of the projected points and the positions of the 
corresponding features in the images. 

3. A method according to claim 2 5 wherein the calculated error is reduced 
by performing processing in accordance with a non-linear least squares 
minimisation procedure. 

4. A method according to claim 3, wherein the calculated error is reduced 
by performing processing in accordance with a Levenberg-Marquardt 
minimisation procedure. 

5. A method according to claim 1, wherein the changed positions for the 
3D feature points and the changed camera projections are calculated by 
decomposing a matrix representing the positions in the images in the subset of 
the features which correspond to the 3D feature points to be changed into the 
product of a first matrix representing the changed camera projections of the 



104 



CFP1793US (2636550) 



images in the subset and a second matrix representing the changed positions 
of the 3D feature points. 

6. A method according to claim 1, wherein the selected 3D feature points 
used to calculate an error comprise every 3D feature point which corresponds 
to a feature having a measured position in at least one of the images in the 
subset being processed. 

7. A method according to claim 1 , wherein the 3 D feature points for which 
changed positions are calculated comprise every 3D feature point which has 
a measured position in at least one of the images in the subset being processed 
but no measured position in any of the other images in the sequence. 

8. A method according to claim 1, wherein each respective subset contains 
the same number of images. 

9. A method according to claim 1, wherein the number of images in a 
subset is set in dependence upon the number of features in the images having 
a position defined in the data to be processed. 

10. A method according to claim 1, wherein the respective subsets of 
images are selected sequentially from the sequence with an overlap. 

11. A method according to claim 1, further comprising the step of 
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generating a signal conveying the further estimates of the camera projections. 

12. A method according to claim 11, further comprising the step of 
recording the signal either directly or indirectly. 

13. A method according to claim 1, further comprising the step of 
processing image data defining the images in the sequence to generate the 
input data. 

14. A method according to claim 1, further comprising the step of using the 
further estimates of the camera projections to generate data defining a 3D 
computer model of the scene in the images. 

15. A method according to claim 14, further comprising the step of 
generating a signal conveying the 3D computer model. 

16. A method according to claim 15, further comprising the step of 
recording the signal either directly or indirectly. 

1 7. Apparatus for processing input data defining (i) the positions of features 
in a sequence of images of at least one object which represent features on the 
object, (ii) an estimate of a respective camera projection for each image 
defining the projection of points on the object into the image, and (iii) 3D 
feature points comprising estimates of the positions in three-dimensions of 
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features on the object represented by the features in the images, to generate 
output data defining further estimates of the camera projections, comprising 
a processor for processing respective subsets of the images in the sequence by: 

calculating an error for the camera projections of the images in the 
subset by projecting selected 3D feature points into the images in the subset 
using the camera projections of the images in the subset and determining the 
distance between the positions of the projected points and the positions of the 
corresponding features in the images; and 

calculating changed positions for at least some of the selected 3D 
feature points and changed camera projections for the images in the subset by 
using the positions in the images in the subset of the features which 
correspond to the selected 3D feature points to determine changed positions 
for at least some of the selected 3D feature points and changed camera 
projections for the images in the subset which reduce the calculated error; 

wherein the processor is arranged to process respective subsets of 
images from the sequence sequentially such that each image in the sequence 
is processed in a subset at least once. 

18. Apparatus according to claim 17, wherein the processor is arranged to 
calculate the changed positions for the 3D feature points and the changed 
camera positions by reducing the calculated error by performing processing in 
accordance with a minimisation procedure to change iteratively the positions 
of at least some of the selected 3D feature points and the camera projections 
of the images in the subset and to calculate an updated error by projecting the 
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3D feature points from their changed positions into the images in the subset 
using the changed camera projections of the images in the subset, and 
determining the distance between the positions of the projected points and the 
positions of the corresponding features in the images. 

19. Apparatus according to claim 18, wherein the processor is arranged to 
reduce the calculated error by performing processing in accordance with a 
non-linear least squares minimisation procedure. 

20. Apparatus according to claim 19, wherein the processor is arranged to 
reduce the calculated error by performing processing in accordance with a 
Levenberg-Marquardt minimisation procedure. 

21. Apparatus according to claim 17, wherein the processor is arranged to 
calculate the changed positions for the 3D feature points and the changed 
camera projections by decomposing a matrix representing the positions in the 
images in the subset of the features which correspond to the 3D feature points 
to be changed into the product of a first matrix representing the changed 
camera projections of the images in the subset and a second matrix 
representing the changed positions of the 3D feature points. 

22. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that the selected 3D feature points used to calculate an 
error comprise every 3D feature point which corresponds to a feature having 
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a measured position in at least one of the images in the subset being processed. 

23. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that the 3D feature points for which changed positions 
are calculated comprise every 3D feature point which has a measured position 
in at least one of the images in the subset being processed but no measured 
position in any of the other images in the sequence. 

24. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that each respective subset contains the same number 
of images. 

25. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that the number of images in a subset is set in 
dependence upon the number of features in the images having a position 
defined in the data to be processed. 

26. Apparatus according to claim 17, wherein the processor is arranged to 
perform processing so that the respective subsets of images are selected 
sequentially from the sequence with an overlap. 

27. Apparatus according to claim 17, further comprising an input data 
generator for processing image data defining the images in the sequence to 
generate the input data. 
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28. Apparatus according to claim 17, further comprising a 3D computer 
model data generator for using the further estimates of the camera projections 
to generate data defining a 3D computer model of the scene in the images. 

29. A storage device storing instructions for causing a programmable 
processing apparatus to become operable to perform a method as set out in at 
least one of claims 1 to 16. 

30. A signal conveying instructions for causing a programmable processing 
apparatus to become operable to perform a method as set out in at least one of 
claims 1 to 16. 

31. In an image processing method for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional 
computer model of the object by matching features in the images, using the 
matching features to determine camera projections defining the projection of 
the object surface into the images, and using the matching features and the 
calculated camera projections to generate data defining the three-dimensional 
computer model of the object, an improvement comprising performing the step 
of using the matching features to determine the camera projections by 
processing data defining (i) the positions of features in a sequence of images 
of the object which represent features on the object, (ii) an estimate of a 
respective camera projection for each image defining the projection of the 
object into the image, and (iii) 3D feature points comprising estimates of the 
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positions in three-dimensions of the features on the object represented by the 
features in the images, to generate data defining further estimates of the 
camera projections, the processing comprising processing respective subsets 
of the images in the sequence such that: 

an error is calculated for the camera projections of the images in the 
subset by projecting selected 3D feature points into the images in the subset 
using the camera projections of the images in the subset to give projected 
points and determining the distance between the positions of the projected 
points and the positions of the corresponding features in the images; and 

changed positions for at least some of the selected 3D feature points 
and changed camera projections for the images in the subset are calculated by 
using the positions in the images in the subset of the features which 
correspond to the selected 3D feature points to determine changed positions 
for at least some of the selected 3D feature points and changed camera 
projections for the images in the subset which reduce the calculated error; 

wherein the respective subsets of images are selected sequentially from 
the sequence so that each image in the sequence is processed in a subset at 
least once. 

32. In an image processing apparatus for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional 
computer model of the object by matching features in the images, using the 
matching features to determine camera projections defining the projection of 
the object surface into the images, and using the matching features and the 
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calculated camera projections to generate data defining the three-dimensional 
computer model of the object, a method of performing the processing to 
determine the camera projections by processing data defining (i) the positions 
of features in a sequence of images of at least the object which represent 
features on the object (ii) an estimate of a respective camera projection for 
each image defining the projection of the object surface into the image, and 
(iii) 3D feature points comprising estimates of the positions in three- 
dimensions of the features on the object represented by the features in the 
images, to generate data defining improved estimates of the camera 
projections, the processing comprising processing respective subsets of the 
images in the sequence by: 

calculating an error for the camera projections of the images in the 
subset by projecting selected 3D feature points into the images in the subset 
using the camera projections of the images in the subset to give projected 
points and determining the distance between the positions of the projected 
points and the positions of the corresponding features in the images; and 

calculating changed positions for at least some of the selected 3D 
feature points and changed camera projections for the images in the subset by 
using the positions in the images in the subset of the features which 
correspond to the selected 3D feature points to determine changed positions 
for at least some of the selected 3D feature points and changed camera 
projections for the images in the subset which reduce the calculated error; 

wherein the respective subsets of images are selected sequentially from 
the sequence so that each image in the sequence is processed in a subset at 
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least once. 

33. In an image processing apparatus having a processor for processing 
image data defining a sequence of images of at least one object to generate a 
three-dimensional computer model of the object by matching features in the 
images, using the matching features to determine camera projections defining 
the projection of the object surface into the images, and using the matching 
features and the calculated camera projections to generate data defining the 
three-dimensional computer model of the object, an improvement wherein the 
processor is arranged to use the matching features to determine the camera 
projections by processing data defining (i) the positions of features in a 
sequence of images of the object which represent features on the object, (ii) 
an estimate of a respective camera projection for each image defining the 
projection of the object into the image, and (iii) 3D feature points comprising 
estimates of the positions in three-dimensions of the features on the object 
represented by the features in the images, to generate data defining further 
estimates of the camera projections, the processing performed by the processor 
comprising processing respective subsets of the images in the sequence such 
that: 

an error is calculated for the camera projections of the images in the 
subset by projecting selected 3D feature points into the images in the subset 
using the camera projections of the images in the subset to give projected 
points and determining the distance between the positions of the projected 
points and the positions of the corresponding features in the images; and 
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changed positions for at least some of the selected 3D feature points 
and changed camera projections for the images in the subset are calculated by 
using the positions in the images in the subset of the features which 
correspond to the selected 3D feature points to determine changed positions 
for at least some of the selected 3D feature points and changed camera 
projections for the images in the subset which reduce the calculated error; 

wherein the respective subsets of images are selected sequentially from 
the sequence so that each image in the sequence is processed in a subset at 
least once. 

34. Apparatus for processing input data defining (i) the positions of features 
in a sequence of images of at least one object which represent features on the 
object, (ii) an estimate of a respective camera projection for each image 
defining the projection of points on the object into the image, and (iii) 3D 
feature points comprising estimates of the positions in three-dimensions of 
features on the object represented by the features in the images, to generate 
output data defining further estimates of the camera projections, comprising 
processing means for processing respective subsets of the images in the 
sequence by: 

calculating an error for the camera projections of the images in the 
subset by projecting selected 3D feature points into the images in the subset 
using the camera projections of the images in the subset and determining the 
distance between the positions of the projected points and the positions of the 
corresponding features in the images; and 
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calculating changed positions for at least some of the selected 3D 
feature points and changed camera projections for the images in the subset by 
using the positions in the images in the subset of the features which 
correspond to the selected 3D feature points to determine changed positions 
for at least some of the selected 3D feature points and changed camera 
projections for the images in the subset which reduce the calculated error; 

wherein the processing means is arranged to process respective subsets 
of images from the sequence sequentially such that each image in the sequence 
is processed in a subset at least once. 

35. A method of processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining the 
projection of points on the object surface into the image, the method 
comprising: 

a selecting step comprising using the matching features to select a first 
image in the sequence and a second image in the sequence for each of a 
plurality of sets of images to give initial end images in the sequence for each 



a set forming step comprising, for the selected pair of end images for 
each set, using the matching features to calculate a camera projection for one 
of the images in the pair relative to the other image in the pair, to form a set 
thereform; 

a set expanding step comprising increasing the number of images in at 



set; 
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least some of the sets by using the matching features to calculate a respective 
camera projection for each of at least some of the images lying in the sequence 
between end images of a set and each of at least some of the images lying in 
the sequence between consecutive sets, each camera projection being 
calculated relative to the images in a set; and 

a set merging step comprising using the matching features to determine 
the relationship between the camera projections calculated for the different 
sets, thereby to merge the sets. 

36. A method according to claim 35, wherein, in the selecting step, an 
image to form an end image is selected in dependence upon the number of 
matching features in the image which are also present in the preceding end 
image in the sequence. 

37. A method according to claim 35, wherein, in the set expanding step, the 
camera projections for images lying between the end images for a given set are 
calculated before the camera projections for images lying between the given 
set and adjacent sets in the sequence are calculated. 

38. A method according to claim 35, wherein, in the step of calculating 
camera projections for images lying between consecutive sets: 

(i) the selected end images are prioritised in dependence upon an 
accuracy of the calculated camera projections of the end images; 

(ii) the end image with the highest priority is selected; 
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(iii) the camera projection for the image in the sequence adjacent the 
selected end image is calculated relative to the selected end image to form a 
new end image; 

(iv) the priority of the end images is updated in dependence upon the 
accuracy of the camera projection calculated in step (iii); and 

(v) steps (ii) to (iv) are repeated. 

39. A method according to claim 35, wherein pairs of end images selected 
in the selecting step are considered in turn and, for a given pair of end images, 
the set forming step to form a set therefrom and the set expanding step to 
calculate a respective camera projection for at least one image lying in the 
sequence between the pair of end images and at least one image lying in the 
sequence between an end image of the pair and an end image for a consecutive 
set, are performed before performing the set forming step and the set 
expanding step for the next pair of end images to be considered. 

40. A method according to claim 35, further comprising the step of 
generating a signal conveying the calculated camera projections. 

41. A method according to claim 40, further comprising the step of 
recording the signal either directly or indirectly. 

42. A method according to claim 35, wherein the sequence of images is a 
temporal sequence in which the images are arranged in time-of-recording 
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order. 

43. A method according to claim 35, wherein the images comprise video 
images. 

44. A method according to claim 35, further comprising the step of 
processing image data defining the images in the sequence to generate the data 
defining the matching features. 

45. A method according to claim 35, further comprising the step of using 
the matching features and the calculated camera projections to generate data 
defining a 3D computer model of the object in the images. 

46. A method according to claim 45, further comprising the step of 
generating a signal conveying the 3D computer model. 

47. A method according to claim 46, further comprising the step of 
recording the signal either directly or indirectly. 

48. A method of processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining a projection 
of the object into the image, the method comprising: 

selecting a plurality of groups of the images, each group comprising at 
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least two images and at least one pair of consecutive groups in the sequence 
having images in the sequence therebetween which are not part of a group; 

forming a set of images from each selected group by calculating a 
respective camera projection for each image in the group relative to a reference 
image in the group; 

calculating a respective camera projection for each of at least some of 
the images in the sequence not selected to form a group relative to a reference 
image in a group; and 

calculating the relationship between the camera projections calculated 
for each group. 

49. Apparatus for processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining the 
projection of points on the object surface into the image, comprising: 

an image selector for using the matching features to select a first image 
in the sequence and a second image in the sequence for each of a plurality of 
sets of images to give initial end images in the sequence for each set; 

a set former for using the matching features to calculate, for the selected 
pair of end images for each set, a camera projection for one of the images in 
the pair relative to the other image in the pair, to form a set thereform; 

a set expander for increasing the number of images in at least some of 
the sets by using the matching features to calculate a respective camera 
projection for each of at least some of the images lying in the sequence 
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between end images of a set and each of at least some of the images lying in 
the sequence between consecutive sets, each camera projection being 
calculated relative to the images in a set; and 

a set merger for using the matching features to determine the 
relationship between the camera projections calculated for the different sets, 
thereby to merge the sets. 

50. Apparatus according to claim49, wherein the image selector is arranged 
to select an image to form an end image in dependence upon the number of 
matching features in the image which are also present in the preceding end 
image in the sequence. 

51. Apparatus according to claim 49, wherein the set expander is arranged 
to calculate the camera projections for images lying between the end images 
for a given set before calculating the camera projections for images lying 
between the given set and adjacent sets in the sequence. 

52. Apparatus according to claim 49, wherein the set expander is arranged 
to calculate camera projections for images lying between consecutive sets by: 

(i) prioritising the selected end images in dependence upon an accuracy 
of the calculated camera projections of the end images; 

(ii) selecting the end image with the highest priority; 

(iii) calculating the camera projection for the image in the sequence 
adjacent the selected end image relative to the selected end image to form a 
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new end image; 

(iv) updating the priority of the end images in dependence upon the 
accuracy of the camera projection calculated in step (iii); and 

(v) repeating steps (ii) to (iv). 

53 . Apparatus according to claims 49, wherein the apparatus is arranged to 
operate so that pairs of end images selected by the image selector are 
considered in turn and, for a given pair of end images, the set former is 
arranged to form a set therefrom and the set expander is arranged to calculate 
a respective camera projection for at least one image lying in the sequence 
between the pair of end images and at least one image lying in the sequence 
between an end image of the pair and an end image for a consecutive set, 
before forming a set and expanding the set for the next pair of end images to 
be considered. 

54. Apparatus according to claim 49, wherein the sequence of images is a 
temporal sequence in which the images are arranged in time-of-recording 
order. 

55. Apparatus according to claim 49, wherein the images comprise video 
images. 

56. Apparatus according to claim 49, further comprising a feature matcher 
for processing image data defining the images in the sequence to generate the 
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data defining the matching features. 

57. Apparatus according to claim 49, further comprising a 3D computer 
model data generator for using the matching features and the calculated camera 
projections to generate data defining a 3D computer model of the object in the 
images. 

58. Apparatus for processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining a projection 
of the object into the image, comprising: 

an image group calculator for selecting a plurality of groups of the 
images, each group comprising at least two images and at least one pair of 
consecutive groups in the sequence having images in the sequence 
therebetween which are not part of a group; 

an image set calculator for forming a set of images from each selected 
group by calculating a respective camera projection for each image in the 
group relative to a reference image in the group; 

a camera projection calculator for calculating a respective camera 
projection for each of at least some of the images in the sequence not selected 
to form a group relative to a reference image in a group; and 

a camera projection relationship calculator for calculating the 
relationship between the camera projections calculated for each group. 
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59. A storage device storing instructions for causing a programmable 
processing apparatus to become operable to perform a method as set out in at 
least one of claims 35 to 48. 

60. A signal conveying instructions for causing a programmable processing 
apparatus to become operable to perform a method as set out in at least one of 
claims 35 to 48. 

61. In an image processing method for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional 
computer model of the object by matching features in the images, using the 
matching features to determine camera projections defining the projection of 
the object surface into the images, and using the matching features and the 
calculated camera projections to generate data defining the three-dimensional 
computer model of the object, an improvement comprising performing the step 
of using the matching features to determine the camera projections by: 

using the matching features to select a first image in the sequence and 
a second image in the sequence for each of a plurality of sets of images to give 
initial end images in the sequence for each set; 

for the selected pair of end images for each set, using the matching 
features to calculate a camera projection for one of the images in the pair 
relative to the. other image in the pair, to form a set thereform; 

increasing the number of images in at least some of the sets by using the 
matching features to calculate a respective camera projection for each of at 
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least some of the images lying between pairs of end images in the sequence 
and each of at least some of the images lying between consecutive sets in the 
sequence, each camera projection being calculated relative to the images in a 
set; and 

using the matching features to determine the relationship between the 
camera projections calculated for the different sets, thereby to merge the sets. 

62. In an image processing apparatus for processing image data defining a 
sequence of images of at least one object to generate a three-dimensional 
computer model of the object by matching features in the images, using the 
matching features to determine camera projections defining the projection of 
the object into the images, and using the matching features and the calculated 
camera projections to generate data defining the three-dimensional computer 
model of the object, a method of performing the processing to determine the 
camera projections by: 

selecting a plurality of groups of the images, each group comprising at 
least two images and at least one pair of consecutive groups in the sequence 
having images in the sequence therebetween which are not part of a group; 

forming a set of images from each selected group by calculating a 
respective camera projection for each image in the group relative to a reference 
image in the group; 

calculating a respective camera projection for each of at least some of 
the images in the sequence not selected to form a group relative to a reference 
image in a group; and 
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calculating the relationship between the camera projections calculated 
for each group. 

63. In an image processing apparatus having a processor for processing 
image data defining a sequence of images of at least one object to generate a 
three-dimensional computer model of the object by matching features in the 
images, using the matching features to determine camera projections defining 
the projection of the object surface into the images, and using the matching 
features and the calculated camera projections to generate data defining the 
three-dimensional computer model of the object, an improvement wherein the 
processor is arranged to use the matching features to determine the camera 
projections by: 

using the matching features to select a first image in the sequence and 
a second image in the sequence for each of a plurality of sets of images to give 
initial end images in the sequence for each set; 

for the selected pair of end images for each set, using the matching 
features to calculate a camera projection for one of the images in the pair 
relative to the other image in the pair, to form a set thereform; 

increasing the number of images in at least some of the sets by using the 
matching features to calculate a respective camera projection for each of at 
least some of the images lying between pairs of end images in the sequence 
and each of at least some of the images lying between consecutive sets in the 
sequence, each camera projection being calculated relative to the images in a 
set; and 
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using the matching features to determine the relationship between the 
camera projections calculated for the different sets, thereby to merge the sets. 

64. Apparatus for processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining the 
projection of points on the object surface into the image, comprising: 

selecting means for using the matching features to select a first image 
in the sequence and a second image in the sequence for each of a plurality of 
sets of images to give initial end images in the sequence for each set; 

set forming means for using the matching features to calculate, for the 
selected pair of end images for each set, a camera projection for one of the 
images in the pair relative to the other image in the pair, to form a set 
thereform; 

set expanding means for increasing the number of images in at least 
some of the sets by using the matching features to calculate a respective 
camera projection for each of at least some of the images lying in the sequence 
between end images of a set and each of at least some of the images lying in 
the sequence between consecutive sets, each camera projection being 
calculated relative to the images in a set; and 

set merging means for using the matching features to determine the 
relationship between the camera projections calculated for the different sets, 
thereby to merge the sets. 
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65. Apparatus for processing data defining matching features in images 
within a sequence of images of at least one object to determine a respective 
camera projection for each of at least some of the images defining a projection 
of the object into the image, comprising: 

means for selecting a plurality of groups of the images, each group 
comprising at least two images and at least one pair of consecutive groups in 
the sequence having images in the sequence therebetween which are not part 
of a group; 

means for forming a set of images from each selected group by 
calculating a respective camera projection for each image in the group relative 
to a reference image in the group; 

means for calculating a respective camera projection for each of at least 
some of the images in the sequence not selected to form a group relative to a 
reference image in a group; and 

means for calculating the relationship between the camera projections 
calculated for each group. 



